SherbyNet - /page/wikibook/entry.mu

Micron Document

<!DOCTYPE html>

<title>Hyperparameter optimization</title>

</head>

Hyperparameter optimization

</h1>

</header>

In <a href="Machine_learning" title="Machine learning">machine learning</a>, hyperparameter optimization<a href="#cite_note-1">[1]</a> or tuning is the problem of choosing a set of optimal <a href="Hyperparameter_(machine_learning)" title="Hyperparameter (machine learning)">hyperparameters</a> for a learning algorithm. A hyperparameter is a <a href="Parameter" title="Parameter">parameter</a> whose value is used to control the learning process, which must be configured before the process starts.<a href="#cite_note-2">[2]</a><a href="#cite_note-3">[3]</a>

Hyperparameter optimization determines the set of hyperparameters that yields an optimal model which minimizes a predefined <a href="Loss_function" title="Loss function">loss function</a> on a given <a href="Data_set" title="Data set">data set</a>.<a href="#cite_note-abs1502.02127-4">[4]</a> The objective function takes a set of hyperparameters and returns the associated loss.<a href="#cite_note-abs1502.02127-4">[4]</a> <a href="Cross-validation_(statistics)" title="Cross-validation (statistics)">Cross-validation</a> is often used to estimate this generalization performance, and therefore choose the set of values for hyperparameters that maximize it.<a href="#cite_note-bergstra-5">[5]</a>

<div class="mw-heading mw-heading2"><h2 id="Approaches">Approaches</h2></div>

<div class="mw-heading mw-heading3"><h3 id="Grid_search">Grid search</h3></div>

The traditional method for hyperparameter optimization has been grid search, or a parameter sweep, which is simply an <a href="Brute-force_search" title="Brute-force search">exhaustive searching</a> through a manually specified subset of the hyperparameter space of a learning algorithm. A grid search algorithm must be guided by some performance metric, typically measured by <a href="Cross-validation_(statistics)" title="Cross-validation (statistics)">cross-validation</a> on the training set<a href="#cite_note-6">[6]</a>

or evaluation on a hold-out validation set.<a href="#cite_note-7">[7]</a>

Since the parameter space of a machine learner may include real-valued or unbounded value spaces for certain parameters, manually set bounds and discretization may be necessary before applying grid search.

For example, a typical soft-margin <a href="Support_vector_machine" title="Support vector machine">SVM</a> <a href="Statistical_classification" title="Statistical classification">classifier</a> equipped with an <a href="Radial_basis_function_kernel" title="Radial basis function kernel">RBF kernel</a> has at least two hyperparameters that need to be tuned for good performance on unseen data: a regularization constant C and a kernel hyperparameter γ. Both parameters are continuous, so to perform grid search, one selects a finite set of "reasonable" values for each, say

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle C\in \{10,100,1000\}}</annotation>

</semantics>

</math><img src="./4124e15320f26a727f12f02d9bc61edc512878fd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:19.462ex; height:2.843ex;" alt="{\displaystyle C\in \{10,100,1000\}}" loading="lazy"></dd>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \gamma \in \{0.1,0.2,0.5,1.0\}}</annotation>

</semantics>

</math><img src="./add7d5ca68cbe82cefb41c6299f04106c03e120f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:21.417ex; height:2.843ex;" alt="{\displaystyle \gamma \in \{0.1,0.2,0.5,1.0\}}" loading="lazy"></dd></dl>

Grid search then trains an SVM with each pair (C, γ) in the <a href="Cartesian_product" title="Cartesian product">Cartesian product</a> of these two sets and evaluates their performance on a held-out validation set (or by internal cross-validation on the training set, in which case multiple SVMs are trained per pair). Finally, the grid search algorithm outputs the settings that achieved the highest score in the validation procedure.

Grid search suffers from the <a href="Curse_of_dimensionality" title="Curse of dimensionality">curse of dimensionality</a>, but is often <a href="Embarrassingly_parallel" title="Embarrassingly parallel">embarrassingly parallel</a> because the hyperparameter settings it evaluates are typically independent of each other.<a href="#cite_note-bergstra-5">[5]</a>

<div class="mw-heading mw-heading3"><h3 id="Random_search">Random search</h3></div>

Random Search replaces the exhaustive enumeration of all combinations by selecting them randomly. This can be simply applied to the discrete setting described above, but also generalizes to continuous and mixed spaces. A benefit over grid search is that random search can explore many more values than grid search could for continuous hyperparameters. It can outperform Grid search, especially when only a small number of hyperparameters affects the final performance of the machine learning algorithm.<a href="#cite_note-bergstra-5">[5]</a> In this case, the optimization problem is said to have a low intrinsic dimensionality.<a href="#cite_note-8">[8]</a> Random Search is also <a href="Embarrassingly_parallel" title="Embarrassingly parallel">embarrassingly parallel</a>, and additionally allows the inclusion of prior knowledge by specifying the distribution from which to sample. Despite its simplicity, random search remains one of the important base-lines against which to compare the performance of new hyperparameter optimization methods.

<div class="mw-heading mw-heading3"><h3 id="Bayesian_optimization">Bayesian optimization</h3></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hatnote{font-style:italic}.mw-parser-output div.hatnote{padding-left:1.6em;margin-bottom:0.5em}.mw-parser-output .hatnote i{font-style:normal}.mw-parser-output .hatnote+link+.hatnote{margin-top:-0.5em}@media print{body.ns-0 .mw-parser-output .hatnote{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div role="note" class="hatnote navigation-not-searchable">Main article: <a href="Bayesian_optimization" title="Bayesian optimization">Bayesian optimization</a></div>

Bayesian optimization is a global optimization method for noisy black-box functions. Applied to hyperparameter optimization, Bayesian optimization builds a probabilistic model of the function mapping from hyperparameter values to the objective evaluated on a validation set. By iteratively evaluating a promising hyperparameter configuration based on the current model, and then updating it, Bayesian optimization aims to gather observations revealing as much information as possible about this function and, in particular, the location of the optimum. It tries to balance exploration (hyperparameters for which the outcome is most uncertain) and exploitation (hyperparameters expected close to the optimum). In practice, Bayesian optimization has been shown<a href="#cite_note-hutter-9">[9]</a><a href="#cite_note-bergstra11-10">[10]</a><a href="#cite_note-snoek-11">[11]</a><a href="#cite_note-thornton-12">[12]</a><a href="#cite_note-krnc-13">[13]</a> to obtain better results in fewer evaluations compared to grid search and random search, due to the ability to reason about the quality of experiments before they are run.

<div class="mw-heading mw-heading3"><h3 id="Gradient-based_optimization">Gradient-based optimization</h3></div>

For specific learning algorithms, it is possible to compute the gradient with respect to hyperparameters and then optimize the hyperparameters using <a href="Gradient_descent" title="Gradient descent">gradient descent</a>. The first usage of these techniques was focused on neural networks.<a href="#cite_note-14">[14]</a> Since then, these methods have been extended to other models such as <a href="Support_vector_machine" title="Support vector machine">support vector machines</a><a href="#cite_note-15">[15]</a> or logistic regression.<a href="#cite_note-16">[16]</a>

A different approach in order to obtain a gradient with respect to hyperparameters consists in differentiating the steps of an iterative optimization algorithm using <a href="Automatic_differentiation" title="Automatic differentiation">automatic differentiation</a>.<a href="#cite_note-17">[17]</a><a href="#cite_note-abs1502.03492-18">[18]</a><a href="#cite_note-19">[19]</a><a href="#cite_note-20">[20]</a> A more recent work along this direction uses the <a href="Implicit_function_theorem" title="Implicit function theorem">implicit function theorem</a> to calculate hypergradients and proposes a stable approximation of the inverse Hessian. The method scales to millions of hyperparameters and requires constant memory.<a href="#cite_note-21">[21]</a>

In a different approach,<a href="#cite_note-22">[22]</a> a hypernetwork is trained to approximate the best response function. One of the advantages of this method is that it can handle discrete hyperparameters as well. Self-tuning networks<a href="#cite_note-23">[23]</a> offer a memory efficient version of this approach by choosing a compact representation for the hypernetwork. More recently, Δ-STN<a href="#cite_note-24">[24]</a> has improved this method further by a slight reparameterization of the hypernetwork which speeds up training. Δ-STN also yields a better approximation of the best-response Jacobian by linearizing the network in the weights, hence removing unnecessary nonlinear effects of large changes in the weights.

Apart from hypernetwork approaches, gradient-based methods can be used to optimize discrete hyperparameters also by adopting a continuous relaxation of the parameters.<a href="#cite_note-25">[25]</a> Such methods have been extensively used for the optimization of architecture hyperparameters in <a href="Neural_architecture_search" title="Neural architecture search">neural architecture search</a>.

<div class="mw-heading mw-heading3"><h3 id="Evolutionary_optimization">Evolutionary optimization</h3></div>

<div role="note" class="hatnote navigation-not-searchable">Main article: <a href="Evolutionary_algorithm" title="Evolutionary algorithm">Evolutionary algorithm</a></div>

Evolutionary optimization is a methodology for the global optimization of noisy black-box functions. In hyperparameter optimization, evolutionary optimization uses <a href="Evolutionary_algorithms" class="mw-redirect" title="Evolutionary algorithms">evolutionary algorithms</a> to search the space of hyperparameters for a given algorithm.<a href="#cite_note-bergstra11-10">[10]</a> Evolutionary hyperparameter optimization follows a <a href="Evolutionary_algorithm#Implementation" title="Evolutionary algorithm">process</a> inspired by the biological concept of <a href="Evolution" title="Evolution">evolution</a>:

<ol><li>Create an initial population of random solutions (i.e., randomly generate tuples of hyperparameters, typically 100+)</li>

<li>Evaluate the hyperparameter tuples and acquire their <a href="Fitness_function" title="Fitness function">fitness function</a> (e.g., 10-fold <a href="Cross-validation_(statistics)" title="Cross-validation (statistics)">cross-validation</a> accuracy of the machine learning algorithm with those hyperparameters)</li>

<li>Rank the hyperparameter tuples by their relative fitness</li>

<li>Replace the worst-performing hyperparameter tuples with new ones generated via <a href="Crossover_(genetic_algorithm)" class="mw-redirect" title="Crossover (genetic algorithm)">crossover</a> and <a href="Mutation_(genetic_algorithm)" class="mw-redirect" title="Mutation (genetic algorithm)">mutation</a></li>

<li>Repeat steps 2-4 until satisfactory algorithm performance is reached or is no longer improving.</li></ol>

Evolutionary optimization has been used in hyperparameter optimization for statistical machine learning algorithms,<a href="#cite_note-bergstra11-10">[10]</a> <a href="Automated_machine_learning" title="Automated machine learning">automated machine learning</a>, typical neural network <a href="#cite_note-kousiouris1-26">[26]</a> and <a href="Deep_learning#Deep_neural_networks" title="Deep learning">deep neural network</a> architecture search,<a href="#cite_note-miikkulainen1-27">[27]</a><a href="#cite_note-jaderberg1-28">[28]</a> as well as training of the weights in deep neural networks.<a href="#cite_note-such1-29">[29]</a>

<div class="mw-heading mw-heading3"><h3 id="Population-based">Population-based</h3></div>

Population Based Training (PBT) learns both hyperparameter values and network weights. Multiple learning processes operate independently, using different hyperparameters. As with evolutionary methods, poorly performing models are iteratively replaced with models that adopt modified hyperparameter values and weights based on the better performers. This replacement model warm starting is the primary differentiator between PBT and other evolutionary methods. PBT thus allows the hyperparameters to evolve and eliminates the need for manual hypertuning. The process makes no assumptions regarding model architecture, loss functions or training procedures.

PBT and its variants are adaptive methods: they update hyperparameters during the training of the models. On the contrary, non-adaptive methods have the sub-optimal strategy to assign a constant set of hyperparameters for the whole training.<a href="#cite_note-30">[30]</a>

<div class="mw-heading mw-heading3"><h3 id="Early_stopping-based">Early stopping-based</h3></div>

A class of early stopping-based hyperparameter optimization algorithms is purpose built for large search spaces of continuous and discrete hyperparameters, particularly when the computational cost to evaluate the performance of a set of hyperparameters is high. Irace implements the iterated racing algorithm, that focuses the search around the most promising configurations, using statistical tests to discard the ones that perform poorly.<a href="#cite_note-irace-31">[31]</a><a href="#cite_note-race-32">[32]</a>

Another early stopping hyperparameter optimization algorithm is successive halving (SHA),<a href="#cite_note-33">[33]</a> which begins as a random search but periodically prunes low-performing models, thereby focusing computational resources on more promising models. Asynchronous successive halving (ASHA)<a href="#cite_note-34">[34]</a> further improves upon SHA's resource utilization profile by removing the need to synchronously evaluate and prune low-performing models. Hyperband<a href="#cite_note-35">[35]</a> is a higher level early stopping-based algorithm that invokes SHA or ASHA multiple times with varying levels of pruning aggressiveness, in order to be more widely applicable and with fewer required inputs.

<div class="mw-heading mw-heading3"><h3 id="Others">Others</h3></div>

<a href="Radial_basis_function" title="Radial basis function">RBF</a><a href="#cite_note-abs1705.08520-36">[36]</a> and <a href="Spectral_method" title="Spectral method">spectral</a><a href="#cite_note-abs1706.00764-37">[37]</a> approaches have also been developed.

<div class="mw-heading mw-heading2"><h2 id="Issues_with_hyperparameter_optimization">Issues with hyperparameter optimization</h2></div>

When hyperparameter optimization is done, the set of hyperparameters are often fitted on a training set and selected based on the generalization performance, or score, of a validation set. However, this procedure is at risk of overfitting the hyperparameters to the validation set. Therefore, the generalization performance score of the validation set (which can be several sets in the case of a cross-validation procedure) cannot be used to simultaneously estimate the generalization performance of the final model. In order to do so, the generalization performance has to be evaluated on a set independent (which has no intersection) of the set (or sets) used for the optimization of the hyperparameters, otherwise the performance might give a value which is too optimistic (too large). This can be done on a second test set, or through an outer <a href="Cross-validation_(statistics)" title="Cross-validation (statistics)">cross-validation</a> procedure called nested cross-validation, which allows an unbiased estimation of the generalization performance of the model, taking into account the bias due to the hyperparameter optimization.

<ul><li><a href="Automated_machine_learning" title="Automated machine learning">Automated machine learning</a></li>

<li><a href="Neural_architecture_search" title="Neural architecture search">Neural architecture search</a></li>

<li><a href="Meta-optimization" title="Meta-optimization">Meta-optimization</a></li>

<li><a href="Model_selection" title="Model selection">Model selection</a></li>

<li><a href="Self-tuning" title="Self-tuning">Self-tuning</a></li>

<li><a href="XGBoost" title="XGBoost">XGBoost</a></li>

<li><a href="Optuna" title="Optuna">Optuna</a></li></ul>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist reflist-columns references-column-width" style="column-width: 30em;">

<li id="cite_note-1"><a href="#cite_ref-1">^</a> Matthias Feurer and Frank Hutter. <a rel="nofollow" class="external text" href="https://link.springer.com/content/pdf/10.1007%2F978-3-030-05318-5_1.pdf">Hyperparameter optimization</a>. In: AutoML: Methods, Systems, Challenges, pages 3–38.

</li>

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite id="CITEREFYang2020" class="citation journal cs1">Yang, Li (2020). "On hyperparameter optimization of machine learning algorithms: Theory and practice". Neurocomputing. 415: 295–316. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/2007.15745">2007.15745</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1016%2Fj.neucom.2020.07.061">10.1016/j.neucom.2020.07.061</a>.</cite>

</li>

<li id="cite_note-3"><a href="#cite_ref-3">^</a> <cite id="CITEREFFranceschiDoniniPerroneKlein2024" class="citation arxiv cs1">Franceschi L, Donini M, Perrone V, Klein A, Archambeau C, Seeger M, Pontil M, Frasconi P (2024). "Hyperparameter Optimization in Machine Learning". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/2410.22854">2410.22854</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/stat.ML">stat.ML</a>].</cite>

</li>

<li id="cite_note-abs1502.02127-4">^ <a href="#cite_ref-abs1502.02127_4-0">a</a> <a href="#cite_ref-abs1502.02127_4-1">b</a> <cite id="CITEREFClaesenBart_De_Moor2015" class="citation arxiv cs1">Claesen, Marc; Bart De Moor (2015). "Hyperparameter Search in Machine Learning". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1502.02127">1502.02127</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</cite>

</li>

<li id="cite_note-bergstra-5">^ <a href="#cite_ref-bergstra_5-0">a</a> <a href="#cite_ref-bergstra_5-1">b</a> <a href="#cite_ref-bergstra_5-2">c</a> <cite id="CITEREFBergstraBengio2012" class="citation journal cs1">Bergstra, James; Bengio, Yoshua (2012). <a rel="nofollow" class="external text" href="http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf">"Random Search for Hyper-Parameter Optimization"</a> (PDF). Journal of Machine Learning Research. 13: 281–305.</cite>

</li>

<li id="cite_note-6"><a href="#cite_ref-6">^</a> Chin-Wei Hsu, Chih-Chung Chang and Chih-Jen Lin (2010). <a rel="nofollow" class="external text" href="http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf">A practical guide to support vector classification</a>. Technical Report, <a href="National_Taiwan_University" title="National Taiwan University">National Taiwan University</a>.

</li>

<li id="cite_note-7"><a href="#cite_ref-7">^</a> <cite id="CITEREFChicco2017" class="citation journal cs1">Chicco D (December 2017). <a rel="nofollow" class="external text" href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5721660">"Ten quick tips for machine learning in computational biology"</a>. BioData Mining. 10 (35): 35. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1186%2Fs13040-017-0155-3">10.1186/s13040-017-0155-3</a>. <a href="PMC_(identifier)" class="mw-redirect" title="PMC (identifier)">PMC</a> <a rel="nofollow" class="external text" href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5721660">5721660</a>. <a href="PMID_(identifier)" class="mw-redirect" title="PMID (identifier)">PMID</a> <a rel="nofollow" class="external text" href="https://pubmed.ncbi.nlm.nih.gov/29234465">29234465</a>.</cite>

</li>

<li id="cite_note-8"><a href="#cite_ref-8">^</a> <cite id="CITEREFZiyuFrankMasrourDavid2016" class="citation journal cs1">Ziyu, Wang; Frank, Hutter; Masrour, Zoghi; David, Matheson; Nando, de Feitas (2016). "Bayesian Optimization in a Billion Dimensions via Random Embeddings". Journal of Artificial Intelligence Research. 55: 361–387. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1301.1942">1301.1942</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1613%2Fjair.4806">10.1613/jair.4806</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:279236">279236</a>.</cite>

</li>

<li id="cite_note-hutter-9"><a href="#cite_ref-hutter_9-0">^</a> <cite id="CITEREFHutterHoosLeyton-Brown2011" class="citation cs2">Hutter, Frank; Hoos, Holger; Leyton-Brown, Kevin (2011), "Sequential Model-Based Optimization for General Algorithm Configuration", <a rel="nofollow" class="external text" href="http://www.cs.ubc.ca/labs/beta/Projects/SMAC/papers/11-LION5-SMAC.pdf">Learning and Intelligent Optimization</a> (PDF), Lecture Notes in Computer Science, vol. 6683, pp. 507–523, <a href="CiteSeerX_(identifier)" class="mw-redirect" title="CiteSeerX (identifier)">CiteSeerX</a> <a rel="nofollow" class="external text" href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.307.8813">10.1.1.307.8813</a>, <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1007%2F978-3-642-25566-3_40">10.1007/978-3-642-25566-3_40</a>, <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-3-642-25565-6</bdi>, <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:6944647">6944647</a></cite>

</li>

<li id="cite_note-bergstra11-10">^ <a href="#cite_ref-bergstra11_10-0">a</a> <a href="#cite_ref-bergstra11_10-1">b</a> <a href="#cite_ref-bergstra11_10-2">c</a> <cite id="CITEREFBergstraBardenetBengioKegl2011" class="citation cs2">Bergstra, James; Bardenet, Remi; Bengio, Yoshua; Kegl, Balazs (2011), <a rel="nofollow" class="external text" href="http://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf">"Algorithms for hyper-parameter optimization"</a> (PDF), Advances in Neural Information Processing Systems</cite>

</li>

<li id="cite_note-snoek-11"><a href="#cite_ref-snoek_11-0">^</a> <cite id="CITEREFSnoekLarochelleAdams2012" class="citation journal cs1">Snoek, Jasper; Larochelle, Hugo; Adams, Ryan (2012). <a rel="nofollow" class="external text" href="http://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf">"Practical Bayesian Optimization of Machine Learning Algorithms"</a> (PDF). Advances in Neural Information Processing Systems. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1206.2944">1206.2944</a>. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2012arXiv1206.2944S">2012arXiv1206.2944S</a>.</cite>

</li>

<li id="cite_note-thornton-12"><a href="#cite_ref-thornton_12-0">^</a> <cite id="CITEREFThorntonHutterHoosLeyton-Brown2013" class="citation journal cs1">Thornton, Chris; Hutter, Frank; Hoos, Holger; Leyton-Brown, Kevin (2013). <a rel="nofollow" class="external text" href="http://www.cs.ubc.ca/labs/beta/Projects/autoweka/papers/autoweka.pdf">"Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms"</a> (PDF). Knowledge Discovery and Data Mining. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1208.3719">1208.3719</a>. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2012arXiv1208.3719T">2012arXiv1208.3719T</a>.</cite>

</li>

<li id="cite_note-krnc-13"><a href="#cite_ref-krnc_13-0">^</a> <cite id="CITEREFKernc2024" class="citation cs2">Kernc (2024), <a rel="nofollow" class="external text" href="https://zenodo.org/records/14461363">SAMBO: Sequential And Model-Based Optimization: Efficient global optimization in Python</a>, <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.5281%2Fzenodo.14461363">10.5281/zenodo.14461363</a>, retrieved 2025-01-30</cite>

</li>

<li id="cite_note-14"><a href="#cite_ref-14">^</a> <cite id="CITEREFLarsenHansenSvarerOhlsson1996" class="citation book cs1">Larsen, Jan; Hansen, Lars Kai; Svarer, Claus; Ohlsson, M (1996). <a rel="nofollow" class="external text" href="http://orbit.dtu.dk/files/4545571/Svarer.pdf">"Design and regularization of neural networks: The optimal use of a validation set"</a> (PDF). Neural Networks for Signal Processing VI. Proceedings of the 1996 IEEE Signal Processing Society Workshop. pp. 62–71. <a href="CiteSeerX_(identifier)" class="mw-redirect" title="CiteSeerX (identifier)">CiteSeerX</a> <a rel="nofollow" class="external text" href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.415.3266">10.1.1.415.3266</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FNNSP.1996.548336">10.1109/NNSP.1996.548336</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>0-7803-3550-3</bdi>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:238874">238874</a>.</cite>

</li>

<li id="cite_note-15"><a href="#cite_ref-15">^</a> <cite id="CITEREFOlivier_ChapelleVladimir_VapnikOlivier_BousquetSayan_Mukherjee2002" class="citation journal cs1">Olivier Chapelle; Vladimir Vapnik; Olivier Bousquet; Sayan Mukherjee (2002). <a rel="nofollow" class="external text" href="http://www.chapelle.cc/olivier/pub/mlj02.pdf">"Choosing multiple parameters for support vector machines"</a> (PDF). Machine Learning. 46: 131–159. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1023%2Fa%3A1012450327387">10.1023/a:1012450327387</a>.</cite>

</li>

<li id="cite_note-16"><a href="#cite_ref-16">^</a> <cite id="CITEREFChuong_BChuan-Sheng_FooAndrew_Y_Ng2008" class="citation journal cs1">Chuong B; Chuan-Sheng Foo; Andrew Y Ng (2008). <a rel="nofollow" class="external text" href="http://papers.nips.cc/paper/3286-efficient-multiple-hyperparameter-learning-for-log-linear-models.pdf">"Efficient multiple hyperparameter learning for log-linear models"</a> (PDF). Advances in Neural Information Processing Systems. 20.</cite>

</li>

<li id="cite_note-17"><a href="#cite_ref-17">^</a> <cite id="CITEREFDomke2012" class="citation journal cs1">Domke, Justin (2012). <a rel="nofollow" class="external text" href="https://web.archive.org/web/20140124182520/http://jmlr.org/proceedings/papers/v22/domke12/domke12.pdf">"Generic Methods for Optimization-Based Modeling"</a> (PDF). Aistats. 22. Archived from <a rel="nofollow" class="external text" href="http://www.jmlr.org/proceedings/papers/v22/domke12/domke12.pdf">the original</a> (PDF) on 2014-01-24. Retrieved 2017-12-09.</cite>

</li>

<li id="cite_note-abs1502.03492-18"><a href="#cite_ref-abs1502.03492_18-0">^</a> <cite id="CITEREFMaclaurinDuvenaudAdams2015" class="citation arxiv cs1">Maclaurin, Dougal; Duvenaud, David; Adams, Ryan P. (2015). "Gradient-based Hyperparameter Optimization through Reversible Learning". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1502.03492">1502.03492</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/stat.ML">stat.ML</a>].</cite>

</li>

<li id="cite_note-19"><a href="#cite_ref-19">^</a> <cite id="CITEREFFranceschiDoniniFrasconiPontil2017" class="citation journal cs1">Franceschi, Luca; Donini, Michele; Frasconi, Paolo; Pontil, Massimiliano (2017). <a rel="nofollow" class="external text" href="http://proceedings.mlr.press/v70/franceschi17a/franceschi17a-supp.pdf">"Forward and Reverse Gradient-Based Hyperparameter Optimization"</a> (PDF). Proceedings of the 34th International Conference on Machine Learning. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1703.01785">1703.01785</a>. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2017arXiv170301785F">2017arXiv170301785F</a>.</cite>

</li>

<li id="cite_note-20"><a href="#cite_ref-20">^</a> <cite id="CITEREFShabanChengHatchBoots2018" class="citation arxiv cs1">Shaban, Amirreza; Cheng, Ching-An; Hatch, Nathan; Boots, Byron (2018). "Truncated Back-propagation for Bilevel Optimization". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1810.10667">1810.10667</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</cite>

</li>

<li id="cite_note-21"><a href="#cite_ref-21">^</a> <cite id="CITEREFLorraineVicolDuvenaud2019" class="citation arxiv cs1">Lorraine, Jonathan; Vicol, Paul; Duvenaud, David (2019). "Optimizing Millions of Hyperparameters by Implicit Differentiation". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1911.02590">1911.02590</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</cite>

</li>

<li id="cite_note-22"><a href="#cite_ref-22">^</a> <cite id="CITEREFLorraineDuvenaud2018" class="citation arxiv cs1">Lorraine, Jonathan; Duvenaud, David (2018). "Stochastic Hyperparameter Optimization through Hypernetworks". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1802.09419">1802.09419</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</cite>

</li>

<li id="cite_note-23"><a href="#cite_ref-23">^</a> <cite id="CITEREFMacKayVicolLorraineDuvenaud2019" class="citation arxiv cs1">MacKay, Matthew; Vicol, Paul; Lorraine, Jon; Duvenaud, David; Grosse, Roger (2019). "Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1903.03088">1903.03088</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</cite>

</li>

<li id="cite_note-24"><a href="#cite_ref-24">^</a> <cite id="CITEREFBaeGrosse2020" class="citation arxiv cs1">Bae, Juhan; Grosse, Roger (2020). "Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/2010.13514">2010.13514</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</cite>

</li>

<li id="cite_note-25"><a href="#cite_ref-25">^</a> <cite id="CITEREFLiuSimonyanYang2018" class="citation arxiv cs1">Liu, Hanxiao; Simonyan, Karen; Yang, Yiming (2018). "DARTS: Differentiable Architecture Search". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1806.09055">1806.09055</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</cite>

</li>

<li id="cite_note-kousiouris1-26"><a href="#cite_ref-kousiouris1_26-0">^</a> <cite id="CITEREFKousiourisCuccinottaVarvarigou2011" class="citation journal cs1">Kousiouris G, Cuccinotta T, Varvarigou T (2011). <a rel="nofollow" class="external text" href="https://www.sciencedirect.com/science/article/abs/pii/S0164121211000951">"The effects of scheduling, workload type and consolidation scenarios on virtual machine performance and their prediction through optimized artificial neural networks"</a>. Journal of Systems and Software. 84 (8): 1270–1291. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1016%2Fj.jss.2011.04.013">10.1016/j.jss.2011.04.013</a>. <a href="Hdl_(identifier)" class="mw-redirect" title="Hdl (identifier)">hdl</a>:<a rel="nofollow" class="external text" href="https://hdl.handle.net/11382%2F361472">11382/361472</a>.</cite>

</li>

<li id="cite_note-miikkulainen1-27"><a href="#cite_ref-miikkulainen1_27-0">^</a> <cite id="CITEREFMiikkulainenLiangMeyersonRawal2017" class="citation arxiv cs1">Miikkulainen R, Liang J, Meyerson E, Rawal A, Fink D, Francon O, Raju B, Shahrzad H, Navruzyan A, Duffy N, Hodjat B (2017). "Evolving Deep Neural Networks". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1703.00548">1703.00548</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.NE">cs.NE</a>].</cite>

</li>

<li id="cite_note-jaderberg1-28"><a href="#cite_ref-jaderberg1_28-0">^</a> <cite id="CITEREFJaderbergDalibardOsinderoCzarnecki2017" class="citation arxiv cs1">Jaderberg M, Dalibard V, Osindero S, Czarnecki WM, Donahue J, Razavi A, Vinyals O, Green T, Dunning I, Simonyan K, Fernando C, Kavukcuoglu K (2017). "Population Based Training of Neural Networks". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1711.09846">1711.09846</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</cite>

</li>

<li id="cite_note-such1-29"><a href="#cite_ref-such1_29-0">^</a> <cite id="CITEREFSuchMadhavanContiLehman2017" class="citation arxiv cs1">Such FP, Madhavan V, Conti E, Lehman J, Stanley KO, Clune J (2017). "Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1712.06567">1712.06567</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.NE">cs.NE</a>].</cite>

</li>

<li id="cite_note-30"><a href="#cite_ref-30">^</a> <cite id="CITEREFLiSpyraPerelDalibard2019" class="citation arxiv cs1">Li, Ang; Spyra, Ola; Perel, Sagi; Dalibard, Valentin; Jaderberg, Max; Gu, Chenjie; Budden, David; Harley, Tim; Gupta, Pramod (2019-02-05). "A Generalized Framework for Population Based Training". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1902.01894">1902.01894</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.AI">cs.AI</a>].</cite>

</li>

<li id="cite_note-irace-31"><a href="#cite_ref-irace_31-0">^</a> <cite id="CITEREFLópez-IbáñezDubois-LacostePérez_CáceresStützle2016" class="citation journal cs1">López-Ibáñez, Manuel; Dubois-Lacoste, Jérémie; Pérez Cáceres, Leslie; Stützle, Thomas; Birattari, Mauro (2016). <a rel="nofollow" class="external text" href="https://doi.org/10.1016%2Fj.orp.2016.09.002">"The irace package: Iterated Racing for Automatic Algorithm Configuration"</a>. Operations Research Perspective. 3 (3): 43–58. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1016%2Fj.orp.2016.09.002">10.1016/j.orp.2016.09.002</a>. <a href="Hdl_(identifier)" class="mw-redirect" title="Hdl (identifier)">hdl</a>:<a rel="nofollow" class="external text" href="https://hdl.handle.net/10419%2F178265">10419/178265</a>.</cite>

</li>

<li id="cite_note-race-32"><a href="#cite_ref-race_32-0">^</a> <cite id="CITEREFBirattariStützlePaqueteVarrentrapp2002" class="citation journal cs1">Birattari, Mauro; Stützle, Thomas; Paquete, Luis; Varrentrapp, Klaus (2002). "A Racing Algorithm for Configuring Metaheuristics". Gecco 2002: 11–18.</cite>

</li>

<li id="cite_note-33"><a href="#cite_ref-33">^</a> <cite id="CITEREFJamiesonTalwalkar2015" class="citation arxiv cs1">Jamieson, Kevin; Talwalkar, Ameet (2015-02-27). "Non-stochastic Best Arm Identification and Hyperparameter Optimization". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1502.07943">1502.07943</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</cite>

</li>

<li id="cite_note-34"><a href="#cite_ref-34">^</a> <cite id="CITEREFLiJamiesonRostamizadehGonina2020" class="citation arxiv cs1">Li, Liam; Jamieson, Kevin; Rostamizadeh, Afshin; Gonina, Ekaterina; Hardt, Moritz; Recht, Benjamin; Talwalkar, Ameet (2020-03-16). "A System for Massively Parallel Hyperparameter Tuning". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1810.05934v5">1810.05934v5</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</cite>

</li>

<li id="cite_note-35"><a href="#cite_ref-35">^</a> <cite id="CITEREFLiJamiesonDeSalvoRostamizadeh2020" class="citation journal cs1">Li, Lisha; Jamieson, Kevin; DeSalvo, Giulia; Rostamizadeh, Afshin; Talwalkar, Ameet (2020-03-16). "Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization". Journal of Machine Learning Research. 18: 1–52. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1603.06560">1603.06560</a>.</cite>

</li>

<li id="cite_note-abs1705.08520-36"><a href="#cite_ref-abs1705.08520_36-0">^</a> <cite id="CITEREFDiazFokoueNanniciniSamulowitz2017" class="citation arxiv cs1">Diaz, Gonzalo; Fokoue, Achille; Nannicini, Giacomo; Samulowitz, Horst (2017). "An effective algorithm for hyperparameter optimization of neural networks". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1705.08520">1705.08520</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.AI">cs.AI</a>].</cite>

</li>

<li id="cite_note-abs1706.00764-37"><a href="#cite_ref-abs1706.00764_37-0">^</a> <cite id="CITEREFHazanKlivansYuan2017" class="citation arxiv cs1">Hazan, Elad; Klivans, Adam; Yuan, Yang (2017). "Hyperparameter Optimization: A Spectral Approach". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1706.00764">1706.00764</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</cite>

</li>

</ol></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hlist dl,.mw-parser-output .hlist ol,.mw-parser-output .hlist ul{margin:0;padding:0}.mw-parser-output .hlist dd,.mw-parser-output .hlist dt,.mw-parser-output .hlist li{margin:0;display:inline}.mw-parser-output .hlist.inline,.mw-parser-output .hlist.inline dl,.mw-parser-output .hlist.inline ol,.mw-parser-output .hlist.inline ul,.mw-parser-output .hlist dl dl,.mw-parser-output .hlist dl ol,.mw-parser-output .hlist dl ul,.mw-parser-output .hlist ol dl,.mw-parser-output .hlist ol ol,.mw-parser-output .hlist ol ul,.mw-parser-output .hlist ul dl,.mw-parser-output .hlist ul ol,.mw-parser-output .hlist ul ul{display:inline}.mw-parser-output .hlist .mw-empty-li{display:none}.mw-parser-output .hlist dt::after{content:": "}.mw-parser-output .hlist dd::after,.mw-parser-output .hlist li::after{content:" · ";font-weight:bold}.mw-parser-output .hlist dd:last-child::after,.mw-parser-output .hlist dt:last-child::after,.mw-parser-output .hlist li:last-child::after{content:none}.mw-parser-output .hlist dd dd:first-child::before,.mw-parser-output .hlist dd dt:first-child::before,.mw-parser-output .hlist dd li:first-child::before,.mw-parser-output .hlist dt dd:first-child::before,.mw-parser-output .hlist dt dt:first-child::before,.mw-parser-output .hlist dt li:first-child::before,.mw-parser-output .hlist li dd:first-child::before,.mw-parser-output .hlist li dt:first-child::before,.mw-parser-output .hlist li li:first-child::before{content:" (";font-weight:normal}.mw-parser-output .hlist dd dd:last-child::after,.mw-parser-output .hlist dd dt:last-child::after,.mw-parser-output .hlist dd li:last-child::after,.mw-parser-output .hlist dt dd:last-child::after,.mw-parser-output .hlist dt dt:last-child::after,.mw-parser-output .hlist dt li:last-child::after,.mw-parser-output .hlist li dd:last-child::after,.mw-parser-output .hlist li dt:last-child::after,.mw-parser-output .hlist li li:last-child::after{content:")";font-weight:normal}.mw-parser-output .hlist ol{counter-reset:listitem}.mw-parser-output .hlist ol>li{counter-increment:listitem}.mw-parser-output .hlist ol>li::before{content:" "counter(listitem)"\a0 "}.mw-parser-output .hlist dd ol>li:first-child::before,.mw-parser-output .hlist dt ol>li:first-child::before,.mw-parser-output .hlist li ol>li:first-child::before{content:" ("counter(listitem)"\a0 "}

/* end https://en.wikipedia.org/ */

</style><style data-mw-deduplicate="TemplateStyles:r1236075235">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbox{box-sizing:border-box;border:1px solid #a2a9b1;width:100%;clear:both;font-size:88%;text-align:center;padding:1px;margin:1em auto 0}.mw-parser-output .navbox .navbox{margin-top:0}.mw-parser-output .navbox+.navbox,.mw-parser-output .navbox+.navbox-styles+.navbox{margin-top:-1px}.mw-parser-output .navbox-inner,.mw-parser-output .navbox-subgroup{width:100%}.mw-parser-output .navbox-group,.mw-parser-output .navbox-title,.mw-parser-output .navbox-abovebelow{padding:0.25em 1em;line-height:1.5em;text-align:center}.mw-parser-output .navbox-group{white-space:nowrap;text-align:right}.mw-parser-output .navbox,.mw-parser-output .navbox-subgroup{background-color:#fdfdfd}.mw-parser-output .navbox-list{line-height:1.5em;border-color:#fdfdfd}.mw-parser-output .navbox-list-with-group{text-align:left;border-left-width:2px;border-left-style:solid}.mw-parser-output tr+tr>.navbox-abovebelow,.mw-parser-output tr+tr>.navbox-group,.mw-parser-output tr+tr>.navbox-image,.mw-parser-output tr+tr>.navbox-list{border-top:2px solid #fdfdfd}.mw-parser-output .navbox-title{background-color:#ccf}.mw-parser-output .navbox-abovebelow,.mw-parser-output .navbox-group,.mw-parser-output .navbox-subgroup .navbox-title{background-color:#ddf}.mw-parser-output .navbox-subgroup .navbox-group,.mw-parser-output .navbox-subgroup .navbox-abovebelow{background-color:#e6e6ff}.mw-parser-output .navbox-even{background-color:#f7f7f7}.mw-parser-output .navbox-odd{background-color:transparent}.mw-parser-output .navbox .hlist td dl,.mw-parser-output .navbox .hlist td ol,.mw-parser-output .navbox .hlist td ul,.mw-parser-output .navbox td.hlist dl,.mw-parser-output .navbox td.hlist ol,.mw-parser-output .navbox td.hlist ul{padding:0.125em 0}.mw-parser-output .navbox .navbar{display:block;font-size:100%}.mw-parser-output .navbox-title .navbar{float:left;text-align:left;margin-right:0.5em}body.skin--responsive .mw-parser-output .navbox-image img{max-width:none!important}@media print{body.ns-0 .mw-parser-output .navbox{display:none!important}}

/* end https://en.wikipedia.org/ */

</style></div><div role="navigation" class="navbox" aria-labelledby="Differentiable_computing254" style="padding:3px"><table class="nowraplinks hlist mw-collapsible autocollapse navbox-inner" style="border-spacing:0;background:transparent;color:inherit"><tbody><tr><th scope="col" class="navbox-title" colspan="2"><style data-mw-deduplicate="TemplateStyles:r1239400231">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbar{display:inline;font-size:88%;font-weight:normal}.mw-parser-output .navbar-collapse{float:left;text-align:left}.mw-parser-output .navbar-boxtext{word-spacing:0}.mw-parser-output .navbar ul{display:inline-block;white-space:nowrap;line-height:inherit}.mw-parser-output .navbar-brackets::before{margin-right:-0.125em;content:"[ "}.mw-parser-output .navbar-brackets::after{margin-left:-0.125em;content:" ]"}.mw-parser-output .navbar li{word-spacing:-0.125em}.mw-parser-output .navbar a>span,.mw-parser-output .navbar a>abbr{text-decoration:inherit}.mw-parser-output .navbar-mini abbr{font-variant:small-caps;border-bottom:none;text-decoration:none;cursor:inherit}.mw-parser-output .navbar-ct-full{font-size:114%;margin:0 7em}.mw-parser-output .navbar-ct-mini{font-size:114%;margin:0 4em}html.skin-theme-clientpref-night .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}@media(prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}}@media print{.mw-parser-output .navbar{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div id="Differentiable_computing254" style="font-size:114%;margin:0 4em">Differentiable computing</div></th></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Differentiable_function" title="Differentiable function">General</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Differentiable_programming" title="Differentiable programming">Differentiable programming</a></li>

<li><a href="Information_geometry" title="Information geometry">Information geometry</a></li>

<li><a href="Statistical_manifold" title="Statistical manifold">Statistical manifold</a></li>

<li><a href="Automatic_differentiation" title="Automatic differentiation">Automatic differentiation</a></li>

<li><a href="Neuromorphic_computing" title="Neuromorphic computing">Neuromorphic computing</a></li>

<li><a href="Pattern_recognition" title="Pattern recognition">Pattern recognition</a></li>

<li><a href="Ricci_calculus" title="Ricci calculus">Ricci calculus</a></li>

<li><a href="Computational_learning_theory" title="Computational learning theory">Computational learning theory</a></li>

<li><a href="Inductive_bias" title="Inductive bias">Inductive bias</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Hardware</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<li><a href="Memristor" title="Memristor">Memristor</a></li>

<li><a href="SpiNNaker" title="SpiNNaker">SpiNNaker</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Software libraries</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="TensorFlow" title="TensorFlow">TensorFlow</a></li>

<li><a href="PyTorch" title="PyTorch">PyTorch</a></li>

<li><a href="Keras" title="Keras">Keras</a></li>

<li><a href="Scikit-learn" title="Scikit-learn">scikit-learn</a></li>

<li><a href="Theano_(software)" title="Theano (software)">Theano</a></li>

<li><a href="MindSpore" title="MindSpore">MindSpore</a></li></ul>

</div></td></tr><tr><td class="navbox-abovebelow" colspan="2"><div>

<ul><li> Portals

<ul><li><a href="Portal%3AComputer_programming" title="Portal:Computer programming">Computer programming</a></li>

<li><a href="Portal%3ATechnology" title="Portal:Technology">Technology</a></li></ul></li></ul>

</div></td></tr></tbody></table></div></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2025-07-10" href="https://en.wikipedia.org/wiki/?title=Hyperparameter_optimization&oldid=1299851399">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>